In this impact evaluation, we study the effect of the Tá de Pé App and Campaign. Tá de Pé (TDP) is an initiative to improve the responsiveness in Education Infrastructure public expenditures. From August 2017 to February 2019, we studied the impact of the App and the impact of an online email campaign designed to pressure all municipalities known to have delayed school constructions according to the Brazilian Federal government Ministry of Education.
Despite being among the ten largest economies in the world, Brazil scores consistently in the last positions at the PISA education exam. Many public school constructions in Brazil are delayed or unfinished due to inefficiencies and corruption. This impacts the long-run productivity and welfare in the country and limits the returns of investments in the country.
We evaluated three interventions:
To access the impact, we randomize a set of schools and municipalities not to receive the intervention. We then studied construction execution statistics, construction status, and construction finishing dates.
The first intervention found that the introduction of the App increased the likelihood of school completion by 6.85 percent. The second intervention found that the App pressure over the bureaucracy increased the investment in the schools by 157.49 percent. The last intervention had a null impact on school constructions statistics.
For further development, we suggest the following:
Gamification of the App, to attract younger users: younger people have more time to use the App. Focusing on this group may improve App usage and the end-line impact on school constructions. For example, a partnership with Pokemon Go, to put Pokemons over the school constructions, together with banners embedded in the game that incentivize reporting, could increase the end-line impact.
Expansion of the intervention scope: school construction is only part of a much broader set of educational services provided. School quality and teacher attendance are also essential and problematic in Brazil. Moreover, the design can be easily transported to other fields, such as health care. This could make the App a comprehensive whistleblowing tool for both education and health care systems.
Shocks in the cost-benefits of using the App: improve the cost-benefit equation could increase the potential number of users. This can be done by providing Tinder, Rappi, Uber, or Google Play credits for users that effectively report.
The Tá de Pé project is an initiative carried out by the Transparência Brasil (Brazilian Transparency) to foster bottom-up accountability. The App was built to facilitate people to go to school construction sites, check their status, and request information to the public authorities about the steps toward finishing the school constructions.
The app works as follows. From a list of constructions available in the Ministry of Education website, the Brazilian Transparency mapped all the schools and made them available in the Ta de Pé App. Any person with the App can go to a construction site and take pictures. The Ta de Pé bot sends the images for engineers to evaluate the execution and suggest whether to report the construction site as delayed. If the construction is reported, then the App sends a request to the mayor’s office asking the public officials to explain why the building is unfinished and requesting better completion estimates.
The App has received the 2016 Google Social Impact grant, with more than 200 thousand popular votes and it has been online since April 2017.
We divided the impact evaluation into three interventions. The first two interventions will evaluate the impact of the App on reported school constructions outcomes. The last intervention will assess the result of a campaign run by the Tá de Pé bot, that reported all the school constructions marked as delayed in the Ministry of Education dataset.
In the first intervention, we use simple randomization, placing 150 municipalities in the control group. In the second and third interventions, we use block randomization, randomizing by three characteristics: Brazilian State, Construction Status (delayed, ongoing, stopped), and above median executed indicator.
To evaluate the random assignment, we used the following pre-treatment variables:
For the first randomization, we used the core R function sample. For the block randomization, we use the package randomizr for R. We take the approach of a small control group, as this was a sensitive condition to not prevent the app from being evaluated. In the first intervention, we select 150 municipalities. In the second and third interventions, we selected around 600 construction sites. The difference in the randomization procedure is due to better information about construction sites during the second and third interventions.
Qualitatively, the intervention in each step consists of the following:
Intervention 1: We selected 344 construction sites to not appear in the app.
Intervention 2: We selected 656 construction sites to not appear in the app.
Intervention 3: We select 523 construction sites to not receive the TDP bot campaign.
We investigated the impact on six outcomes of interest:
Our hypothesis, conditional on the app having a positive effect in terms of welfare, is to find that:
To check the manipulation, we look at the number and places of app download.
For the estimation, we use the following regression equation:
\[ Y_i \ = \ \alpha + \beta T_i + \gamma X_i + \theta Z_i + \varepsilon_i \]
\(i\) indexes the case. \(Y_i\) in an outcome, as described in the previous section. \(\beta\) is the quantity of interest (Average Treatment Effect). \(T_i\) is the treatment that in our case, is an intervention occurrence indicator with two values, zero (no intervention) and one (intervention). \(\gamma\) is a vector of fixed effects, and \(X_i\) is a matrix of Brazilian states’ fixed effects. \(\theta\) is a vector of controls and \(Z_i\) an array of controls for the case \(i\). \(\varepsilon_i\) is the error term.
We cluster the standard errors at the municipal level, as the investment decisions are taken by the mayor’s offices.
As robustness, we fit two extra models. First, we re-run the main model where we have the following: (i) we ran a model without controls or fixed effects; (ii) we run the models adding the control variables, which are the same as the ones used in the covariate balance tests; (iii) we run the regression model with State fixed effects. The results are in the Appendix.
Second, we run the main models using inverse probability weights where we used block randomization and a one-to-one nearest neighborhood matching. We use the control variables as matching characteristics. These robustness checks are intended to correct the imbalance caused by the minimal control group approach that we employed in all three interventions.
In the first intervention, the municipalities in the control group were displayed in the treatment group for about two weeks, in January 2018. When the mistake was detected, the towns were removed promptly. A few municipalities have App activity, but controlling and/or excluding these municipalities from the estimation does not change the results. Note also that this problem lowers the chance to detect an existant effect, making the estimator more conservative.
In the second and third interventions, there were no threats to the identification, with the treatment being fully implemented.
The intervention one was designed to evaluate the impact of launching the first version of the App. The first version was built for Android only, and was launched in August 2017 and lasted until February 2018.
The randomization was performed at the municipal level. We selected 150 municipalities for the control group while the rest were placed in the treatment group. The control group worked by deleting all the construction sites in control assigned cities.
In the next Table, we display the balance test, showing that the covariates are balanced in the treatment and control groups.
##
## --------Summary descriptives table by 'treatOriginal'---------
##
## __________________________________________________________________
## 0 1 p.overall
## N=150 N=1030
## ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
## Log Population (2015) 10.189 (1.079) 10.172 (1.096) 0.858
## Log Poor Families (2010) 7.772 (1.022) 7.780 (1.040) 0.926
## Log Total Transfers (2016) 15.854 (0.832) 15.845 (0.844) 0.903
## IDEB Inicial Years (2015) 4.973 (0.963) 4.970 (0.979) 0.978
## IDEB Final Years (2015) 4.015 (0.837) 3.992 (0.804) 0.757
## ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
As expected, in the treatment and the control, there are no statistically significant differences in terms of Population in 2015, the number of Poor Families in the 2010 IBGE Census, total and FUNDEB transfers to the municipality in 2016, and the IDEB education quality indicators.
Manipulation in experiments is defined as the extent through which the treatment was successfully delivered. In the case of the App intervention, it represents whether having the App improved the results in terms of school construction completion rates. We present two manipulation indicators. First, we look into the municipalities where we had a successful download and use of the App.
Second, we look into App downloads during the intervention 1 period in the following graph.
The map displays a good territorial variability. There were 455 downloads in the 1020 municipalities originally in the dataset at the beginning of the impact evaluation. Downloads peak during the Facebook TDP campaign, right after the App launch in August, then diminishes over time.
In terms of manipulation, there were downloads in 44.6% of the municipalities, and the downloads were temporally concentrated. This indicates that the Facebook campaign worked, but in terms of manipulation, represents a weak treatment delivery. This because the step-by-step usage of the App involves:
As the App download is the first step, this represents low manipulation.
The intervention was sufficient to improve the school completion rate. It increased in 6.85 percent the chance that a school construction becomes reported as finished by the end of the intervention. All other coefficients were not statistically significant at conventional levels.
This finding is positive in the sense that bottom-up pressure exerted by the App worked to increase the completion rate of constructions.
In the randomization, we adopted a minimal control group approach. This has implications for statistical power and sensitivity of the results, as much of the variation is placed on the treatment versus the control group. To increase the reliability of the results, we run the main using one-to-one Nearest Neighborhood Matching with Mahalanobis distance. The findings follow below.
As we can see, the impact of the App on the school construction completion resists the robustness checks. This reinforces the reliability of this effect.
The introduction of App had a positive effect on the school construction completion rates.
In intervention 2, we study the impact of the presence of the App on school construction outcomes. Intervention 2 is similar to intervention 1 in all but three characteristics. First, the TDP App was now available for iOS devices. Second, the randomization is now performed at the school level, blocking by characteristics noted below. Finally, the intervention period is August 2018 to February 2019.
We randomize the App at the school level, placing 15% of schools that could potentially appear in the App in the control group. We used block randomization, blocking by four characteristics:
The blocks and the inverse probability weights are located in the Appendix. The next table shows that the randomization is balanced by the municipal pre-treatment variables.
##
## --------Summary descriptives table by 'treatApp'---------
##
## __________________________________________________________________
## Control Treatment p.overall
## N=654 N=3895
## ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
## Log Population (2015) 10.497 (1.540) 10.501 (1.449) 0.956
## Log Poor Families (2010) 8.092 (1.431) 8.087 (1.319) 0.930
## Log Total Transfers (2016) 16.095 (1.226) 16.087 (1.136) 0.875
## IDEB Inicial Years (2015) 4.886 (0.998) 4.908 (0.977) 0.606
## IDEB Final Years (2015) 3.889 (0.734) 3.891 (0.713) 0.949
## ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
In terms of manipulation, we analyze two outcomes: municipality App downloads and downloads over time.
Downloads by municipality:
## Joining, by = "City"
Downloads over time – Intervention 2:
The total number of municipalities that had downloads of the App is 433. There are two spikes in the usage of the App. One in August, right after the intervention start and a second spike around December. The first spike is a result of a TDP social media campaign for boosting App usage. The second spike is a spillover from the TDP reporting campaign, here known as the intervention 3.
The number of downloads is considerably smaller in this second intervention when compared with intervention 1. As we discussed, this represents a weak manipulation, making detecting treatment effects a hard task.
The full model shows us null effects in all the studied outcomes.
As we used block randomization, the treatment effect is only corrected estimated when we use the inverse probability weights to reweight the outcomes. The results follow in the table below, showing null effects.
Using nearest neighborhood matching the results improve a little, and we can see that the App improved the difference between the baseline and the end-line reported investment by 157.49 percent.
Although there were no completion effects as in intervention 1, we had a substantial impact investment in the nearest neighborhood matched sample.
We have null treatment effects in all but one outcome, in the nearest neighborhood matched sample. In the NN-matching, we find that the App improves the percentage of the executed budget.
Intervention 3 focused on the effect of a TDP Campaign to boost response, in a top-down email campaign to select schools in December 2018.
We randomize the TDP Campaign, placing 15% of schools that could potentially receive the email campaign advocacy in the control group. We used block randomization, blocking by four characteristics:
The blocks and the inverse probability weights are located in the Appendix. The next table shows that the randomization is balanced by the municipal pre-treatment variables.
##
## --------Summary descriptives table by 'treatCampaign'---------
##
## __________________________________________________________________
## Control Treatment p.overall
## N=521 N=3151
## ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
## Log Population (2015) 10.630 (1.488) 10.525 (1.509) 0.137
## Log Poor Families (2010) 8.218 (1.365) 8.145 (1.344) 0.256
## Log Total Transfers (2016) 16.181 (1.163) 16.123 (1.187) 0.288
## IDEB Inicial Years (2015) 4.812 (0.924) 4.783 (0.929) 0.523
## IDEB Final Years (2015) 3.824 (0.684) 3.845 (0.699) 0.581
## ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯
In the manipulation, let us analyze the ex-post effect by looking into responses.
First, we have the following chart for the validity of the responses. We coded the validity of responses in three categories. First, a valid answer provides a minimal explanation that covers the inquiries made by the campaign email. An invalid answer represents an answer without any explanation. Not answer means that the campaign email has not been addressed by the :
As we can see, the state with the most valid responses was the Rio Grande do Sul, while the state with the most significant number of non-response was Maranhao. Maranhao is also the state with most school constructions in the Campaign, which can partially explain their rates. Para is a positive outlier in this analysis, as it has the second-largest number of schools in the campaign, but it managed to answer almost half of the requests with valid answers. Amapa is a negative outlier in the analysis, as no campaign emails have been answered.
The campaign has a higher manipulation impact, which is expected since it consists of sending emails and monitoring responses. One caveat is that the advantage of the App is that it has a more personal component attached to it. Combining these elements could be a good alternative here.
In the model below, we show that the campaign, despite the sizeable manipulation, had null effects on the studied outcomes.
Using inverse probability weights given by the block randomization, we show again that the campaign had null effects on the studied outcomes.
Finally, using the nearest neighborhood matching, we still have a consistent null effect of the campaign on the studied outcomes.
The campaign had a consistent null effect across all treatments. This goes per most email campaigns aimed at improving development outcomes.
In this report, we evaluated the TDP App (interventions 1 and 2) and the TDP campaign (intervention 3). We find that the campaign has a consistent null effect across all the analysis.
In terms of suggestions, for the first two interventions, we strongly suggest measures to improve the manipulation, such as gamification of the App and shocks in the cost-benefit calculations (such as bonuses for using the App, discount coupons in widely used Apps, and others). For the third intervention, a recent paper by Larreguy et al. (2019) showed that repeating Facebook boosting campaign has a strong effect, as saturation is key for achieving positive results in terms of online campaigns. Therefore, repeating the intervention for small periods of time (i.e., monthly, could improve the effects).
In this appendix, we put the APSA Experimental Section report.
Main intervention questions: The experiment studies two problems. First, whether the introduction of the Ta de Pe App improved the governmental school constructions in Brazil. Second, whether a top-down campaign to pressure the bureaucracies to deliver the school infrastructure enhanced outcomes. Therefore, the project here studies how technology facilitates bottom-up and top-down pressure to improve service provision.
Hypothesis: There is a set of findings showing that bottom-up peer pressure improves outcomes in developing economies. The main work in the field shows positive peer-pressure effects:
Bjorkman, Martina and Jakob Svensson. 2009. “Power to the People: Evidence from a randomized field experiment of a community-based monitoring project in Uganda.” Quarterly Journal of Economics 124(2):735–769.
Bjorkman, Martina and Jakob Svensson. 2010. “When is community-based monitoring effective? Evidence from a randomized experiment in primary health in Uganda.” Journal of the European Economic Association 8(2-3):571–581.
Bjorkman Nyqvist, Martina, Damien de Walque and Jakob Svensson. 2017. “Experimental evidence on the long-run impact of community-based monitoring.” American Economic Journal: Applied Economics 9(1):33–69.
However, a set of recent papers set out to recheck these hypotheses, finding an overall null result:
This project is essential to reevaluate this dispute and try to propose new means to understand how bottom-up accountability works. First, we are conducting this experiment in a middle-income country, that within the country has municipalities with social indicators close to developed European countries in some areas and developing economies such as African countries, in some other areas. Second, we are evaluating school constructions, and the presence of schools have essential consequences in terms of long-run economic development, the household income composition, and woman empowerment.
All schools constructions funded by the Brazilian Ministry of Education participated in this study. We had no schools excluded from the program.
The randomization code was:
vectreat <- c(rep(0,150), rep(1, nrow(dat)-150))
AIC_info = numeric()
for (i in 1:5000) {
vectreat <- sample(vectreat, length(vectreat))
mod <- glm(formula(paste('vectreat', vars, sep = '~')),
family = binomial, data = dat)
if(sum(summary(mod)$coefficients[-1,4]<.2)==0) {
dat = data.frame(dat, vtreat = vectreat)
AIC_info = c(AIC_info, AIC(mod))
}
}
And then we checked whether the randomization had one of the covariates selected here significantly predicting the random assignment. We presented a vector of acceptable random assignment in these grounds, and the Brazilian Transparency selected the one we are using in the first intervention.
In interventions 1 and 2, the schools in the control group did not show up in the App. In the intervention 3, the mayors’ office has not received emails from the Ta de Pe bot.
The randomizr code for intervention 2 follows below. OnApp means treatment and OffApp means the control for this intervention:
decl_intheapp <- declare_ra(blocks = dat_intheapp$var_blocking,
prob_each = c(.85,.15),
conditions = c('OnApp', 'OffApp'))
dat_intheapp$Z_intheapp <- block_ra(blocks = dat_intheapp$var_blocking,
conditions = c('OnApp', 'OffApp'),
prob_each = c(.85,.15))
dat_intheapp$IPW_intheapp <- 1/obtain_condition_probabilities(decl_intheapp,
dat_intheapp$Z_intheapp)
The randomizr code for intervention 3 follows below. Campaign means treatment and ControlCampaign means the control for this intervention:
decl_campaign <- declare_ra(blocks = dat_campaign$var_blocking,
prob_each = c(.85,.15),
conditions = c('Campaign', 'ControlCampaign'))
dat_campaign$Z_campaign <- block_ra(blocks = dat_campaign$var_blocking,
conditions = c('Campaign', 'ControlCampaign'),
prob_each = c(.85,.15))
dat_campaign$IPW_campaign <- 1/obtain_condition_probabilities(decl_campaign,
dat_campaign$Z_campaign)
*Block randomization: We used simple randomization in the first assignment. In the second and third interventions, we blocked by the Brazilian State, the construction status, and above the median execution. The summary of the blocks, with the IPWs assigned follow below.
| state | status | above_median_executed | var_blocking | Z_campaign | IPW_campaign | |
|---|---|---|---|---|---|---|
| 1 | CE | Unfinished | AboveMedianExecuted | CE_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 2 | PI | Unfinished | AboveMedianExecuted | PI_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 3 | PI | Unfinished | AboveMedianExecuted | PI_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 4 | MA | Unfinished | AboveMedianExecuted | MA_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 5 | RN | Unfinished | AboveMedianExecuted | RN_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 6 | TO | Unfinished | AboveMedianExecuted | TO_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 7 | TO | Unfinished | Below_Median_Executed | TO_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 8 | SP | Ongoing | AboveMedianExecuted | SP_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 9 | PB | Unfinished | Below_Median_Executed | PB_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 10 | PB | Ongoing | AboveMedianExecuted | PB_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 13 | MG | Unfinished | AboveMedianExecuted | MG_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 14 | GO | Unfinished | AboveMedianExecuted | GO_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 15 | RJ | Unfinished | AboveMedianExecuted | RJ_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 17 | ES | Ongoing | AboveMedianExecuted | ES_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 18 | PI | Ongoing | AboveMedianExecuted | PI_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 19 | CE | Unfinished | AboveMedianExecuted | CE_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 21 | CE | Unfinished | Below_Median_Executed | CE_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 24 | MG | Unfinished | AboveMedianExecuted | MG_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 26 | BA | Unfinished | AboveMedianExecuted | BA_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 30 | RJ | Unfinished | Below_Median_Executed | RJ_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 33 | GO | Ongoing | AboveMedianExecuted | GO_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 36 | PB | Unfinished | Below_Median_Executed | PB_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 37 | RS | Unfinished | Below_Median_Executed | RS_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 39 | RS | Unfinished | AboveMedianExecuted | RS_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 41 | SP | Unfinished | AboveMedianExecuted | SP_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 48 | MS | Unfinished | AboveMedianExecuted | MS_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 50 | MT | Unfinished | AboveMedianExecuted | MT_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 51 | ES | Ongoing | Below_Median_Executed | ES_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 52 | MG | Unfinished | Below_Median_Executed | MG_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 53 | MG | Stopped | AboveMedianExecuted | MG_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 55 | RS | Ongoing | AboveMedianExecuted | RS_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 58 | TO | Unfinished | AboveMedianExecuted | TO_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 59 | PR | Unfinished | Below_Median_Executed | PR_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 61 | PB | Unfinished | AboveMedianExecuted | PB_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 64 | RN | Unfinished | AboveMedianExecuted | RN_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 68 | GO | Unfinished | AboveMedianExecuted | GO_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 73 | PI | Unfinished | Below_Median_Executed | PI_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 74 | MG | Unfinished | Below_Median_Executed | MG_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 84 | MT | Unfinished | AboveMedianExecuted | MT_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 85 | BA | Ongoing | AboveMedianExecuted | BA_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 88 | BA | Ongoing | AboveMedianExecuted | BA_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 89 | BA | Stopped | AboveMedianExecuted | BA_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 90 | PR | Stopped | AboveMedianExecuted | PR_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 94 | PA | Ongoing | AboveMedianExecuted | PA_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 95 | PA | Stopped | AboveMedianExecuted | PA_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 97 | PR | Stopped | AboveMedianExecuted | PR_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 100 | PA | Ongoing | Below_Median_Executed | PA_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 105 | RR | Ongoing | AboveMedianExecuted | RR_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 106 | AM | Unfinished | AboveMedianExecuted | AM_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 108 | AM | Unfinished | Below_Median_Executed | AM_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 112 | AM | Unfinished | Below_Median_Executed | AM_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 116 | PA | Ongoing | AboveMedianExecuted | PA_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 118 | PA | Ongoing | Below_Median_Executed | PA_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 119 | PA | Stopped | AboveMedianExecuted | PA_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 122 | PA | Stopped | Below_Median_Executed | PA_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 129 | CE | Ongoing | AboveMedianExecuted | CE_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 132 | TO | Ongoing | AboveMedianExecuted | TO_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 136 | TO | Stopped | AboveMedianExecuted | TO_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 137 | RR | Stopped | AboveMedianExecuted | RR_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 139 | RR | Stopped | Below_Median_Executed | RR_Stopped_Below_Median_Executed | ControlCampaign | 6.666667 |
| 140 | RR | Stopped | Below_Median_Executed | RR_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 141 | RR | Ongoing | Below_Median_Executed | RR_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 145 | AP | Unfinished | AboveMedianExecuted | AP_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 146 | AP | Unfinished | AboveMedianExecuted | AP_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 149 | AP | Unfinished | Below_Median_Executed | AP_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 153 | AP | Unfinished | Below_Median_Executed | AP_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 157 | GO | Ongoing | Below_Median_Executed | GO_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 158 | GO | Stopped | AboveMedianExecuted | GO_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 163 | GO | Ongoing | Below_Median_Executed | GO_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 166 | GO | Stopped | AboveMedianExecuted | GO_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 169 | GO | Stopped | Below_Median_Executed | GO_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 174 | MA | Ongoing | AboveMedianExecuted | MA_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 175 | MA | Ongoing | Below_Median_Executed | MA_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 176 | MA | Ongoing | Below_Median_Executed | MA_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 186 | MA | Unfinished | Below_Median_Executed | MA_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 198 | SE | Unfinished | Below_Median_Executed | SE_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 202 | RR | Unfinished | AboveMedianExecuted | RR_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 203 | RR | Unfinished | Below_Median_Executed | RR_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 204 | MA | Unfinished | AboveMedianExecuted | MA_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 208 | MT | Ongoing | AboveMedianExecuted | MT_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 210 | AP | Ongoing | Below_Median_Executed | AP_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 228 | MS | Unfinished | Below_Median_Executed | MS_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 231 | PA | Unfinished | AboveMedianExecuted | PA_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 233 | MG | Ongoing | AboveMedianExecuted | MG_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 241 | MS | Unfinished | AboveMedianExecuted | MS_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 242 | MT | Stopped | AboveMedianExecuted | MT_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 247 | AM | Unfinished | AboveMedianExecuted | AM_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 252 | RO | Unfinished | AboveMedianExecuted | RO_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 253 | RO | Ongoing | Below_Median_Executed | RO_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 256 | PI | Ongoing | AboveMedianExecuted | PI_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 265 | RN | Ongoing | AboveMedianExecuted | RN_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 270 | RN | Ongoing | AboveMedianExecuted | RN_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 272 | RN | Stopped | Below_Median_Executed | RN_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 280 | PE | Unfinished | AboveMedianExecuted | PE_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 285 | AL | Ongoing | AboveMedianExecuted | AL_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 287 | SE | Unfinished | AboveMedianExecuted | SE_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 288 | SE | Ongoing | AboveMedianExecuted | SE_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 300 | MG | Ongoing | Below_Median_Executed | MG_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 313 | SP | Unfinished | AboveMedianExecuted | SP_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 316 | RO | Stopped | AboveMedianExecuted | RO_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 319 | PR | Unfinished | AboveMedianExecuted | PR_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 320 | PR | Ongoing | AboveMedianExecuted | PR_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 321 | PR | Unfinished | AboveMedianExecuted | PR_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 327 | RS | Stopped | AboveMedianExecuted | RS_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 328 | MT | Ongoing | Below_Median_Executed | MT_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 329 | MT | Stopped | Below_Median_Executed | MT_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 333 | BA | Ongoing | Below_Median_Executed | BA_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 340 | BA | Unfinished | Below_Median_Executed | BA_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 341 | BA | Unfinished | AboveMedianExecuted | BA_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 357 | PR | Stopped | Below_Median_Executed | PR_Stopped_Below_Median_Executed | ControlCampaign | 6.666667 |
| 359 | PR | Stopped | Below_Median_Executed | PR_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 371 | RS | Ongoing | Below_Median_Executed | RS_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 384 | SP | Ongoing | Below_Median_Executed | SP_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 391 | MG | Ongoing | AboveMedianExecuted | MG_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 410 | SE | Unfinished | AboveMedianExecuted | SE_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 411 | SE | Ongoing | Below_Median_Executed | SE_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 415 | AM | Ongoing | AboveMedianExecuted | AM_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 418 | BA | Stopped | AboveMedianExecuted | BA_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 420 | SP | Unfinished | Below_Median_Executed | SP_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 422 | SC | Ongoing | Below_Median_Executed | SC_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 424 | PE | Ongoing | AboveMedianExecuted | PE_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 439 | GO | Unfinished | Below_Median_Executed | GO_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 445 | MS | Ongoing | Below_Median_Executed | MS_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 446 | MS | Ongoing | AboveMedianExecuted | MS_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 449 | PB | Ongoing | Below_Median_Executed | PB_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 450 | PE | Unfinished | Below_Median_Executed | PE_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 456 | CE | Stopped | AboveMedianExecuted | CE_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 467 | BA | Unfinished | Below_Median_Executed | BA_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 487 | AL | Unfinished | Below_Median_Executed | AL_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 488 | BA | Ongoing | Below_Median_Executed | BA_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 496 | GO | Ongoing | AboveMedianExecuted | GO_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 505 | MG | Ongoing | Below_Median_Executed | MG_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 507 | MG | Stopped | AboveMedianExecuted | MG_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 512 | MT | Ongoing | AboveMedianExecuted | MT_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 515 | PB | Unfinished | AboveMedianExecuted | PB_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 517 | PE | Ongoing | AboveMedianExecuted | PE_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 526 | RN | Stopped | AboveMedianExecuted | RN_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 527 | RS | Ongoing | Below_Median_Executed | RS_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 530 | RS | Ongoing | AboveMedianExecuted | RS_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 531 | SC | Unfinished | AboveMedianExecuted | SC_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 537 | AL | Unfinished | AboveMedianExecuted | AL_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 559 | SC | Ongoing | Below_Median_Executed | SC_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 560 | SC | Ongoing | AboveMedianExecuted | SC_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 561 | SC | Ongoing | AboveMedianExecuted | SC_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 562 | SC | Stopped | AboveMedianExecuted | SC_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 571 | SP | Ongoing | Below_Median_Executed | SP_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 576 | CE | Ongoing | Below_Median_Executed | CE_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 581 | CE | Ongoing | AboveMedianExecuted | CE_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 589 | CE | Unfinished | Below_Median_Executed | CE_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 604 | PR | Ongoing | AboveMedianExecuted | PR_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 625 | MS | Stopped | AboveMedianExecuted | MS_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 627 | MS | Unfinished | Below_Median_Executed | MS_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 637 | PR | Ongoing | Below_Median_Executed | PR_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 658 | PR | Ongoing | Below_Median_Executed | PR_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 664 | PI | Unfinished | Below_Median_Executed | PI_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 668 | RN | Unfinished | Below_Median_Executed | RN_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 671 | MA | Stopped | AboveMedianExecuted | MA_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 673 | RO | Unfinished | Below_Median_Executed | RO_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 674 | SE | Ongoing | Below_Median_Executed | SE_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 702 | ES | Unfinished | AboveMedianExecuted | ES_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 711 | MT | Unfinished | Below_Median_Executed | MT_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 730 | MT | Stopped | AboveMedianExecuted | MT_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 734 | PE | Ongoing | Below_Median_Executed | PE_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 735 | PA | Unfinished | Below_Median_Executed | PA_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 736 | AL | Ongoing | Below_Median_Executed | AL_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 752 | AM | Stopped | AboveMedianExecuted | AM_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 771 | CE | Stopped | AboveMedianExecuted | CE_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 782 | ES | Stopped | AboveMedianExecuted | ES_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 783 | RS | Stopped | AboveMedianExecuted | RS_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 786 | AM | Ongoing | Below_Median_Executed | AM_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 787 | RS | Stopped | Below_Median_Executed | RS_Stopped_Below_Median_Executed | ControlCampaign | 6.666667 |
| 791 | PE | Stopped | AboveMedianExecuted | PE_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 801 | RO | Unfinished | AboveMedianExecuted | RO_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 803 | AL | Stopped | AboveMedianExecuted | AL_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 804 | SP | Stopped | AboveMedianExecuted | SP_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 818 | TO | Stopped | AboveMedianExecuted | TO_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 820 | SP | Stopped | AboveMedianExecuted | SP_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 830 | AL | Unfinished | AboveMedianExecuted | AL_Unfinished_AboveMedianExecuted | Campaign | 1.176471 |
| 834 | RJ | Ongoing | Below_Median_Executed | RJ_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 835 | RJ | Ongoing | Below_Median_Executed | RJ_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 846 | MA | Ongoing | AboveMedianExecuted | MA_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 884 | PB | Stopped | AboveMedianExecuted | PB_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 932 | RO | Ongoing | AboveMedianExecuted | RO_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 935 | RJ | Stopped | AboveMedianExecuted | RJ_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 936 | RJ | Stopped | AboveMedianExecuted | RJ_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 949 | PA | Unfinished | AboveMedianExecuted | PA_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 952 | RS | Stopped | Below_Median_Executed | RS_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 989 | RN | Stopped | AboveMedianExecuted | RN_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 1006 | TO | Ongoing | Below_Median_Executed | TO_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 1013 | RJ | Ongoing | AboveMedianExecuted | RJ_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 1033 | PI | Stopped | AboveMedianExecuted | PI_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 1054 | AL | Stopped | AboveMedianExecuted | AL_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 1086 | MA | Stopped | AboveMedianExecuted | MA_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 1102 | BA | Stopped | Below_Median_Executed | BA_Stopped_Below_Median_Executed | ControlCampaign | 6.666667 |
| 1118 | BA | Stopped | Below_Median_Executed | BA_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 1136 | MS | Stopped | AboveMedianExecuted | MS_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 1147 | PE | Stopped | AboveMedianExecuted | PE_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 1155 | AC | Ongoing | AboveMedianExecuted | AC_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 1158 | AC | Stopped | AboveMedianExecuted | AC_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 1161 | AL | Unfinished | Below_Median_Executed | AL_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 1163 | AL | Ongoing | AboveMedianExecuted | AL_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 1187 | RN | Ongoing | Below_Median_Executed | RN_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 1225 | SP | Stopped | Below_Median_Executed | SP_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 1231 | SP | Ongoing | AboveMedianExecuted | SP_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 1246 | RJ | Stopped | Below_Median_Executed | RJ_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 1264 | ES | Ongoing | Below_Median_Executed | ES_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 1279 | SE | Stopped | AboveMedianExecuted | SE_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 1287 | SE | Stopped | AboveMedianExecuted | SE_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 1313 | PI | Ongoing | Below_Median_Executed | PI_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 1316 | PI | Ongoing | Below_Median_Executed | PI_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 1318 | PI | Stopped | AboveMedianExecuted | PI_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 1338 | RS | Unfinished | AboveMedianExecuted | RS_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 1371 | PE | Unfinished | Below_Median_Executed | PE_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 1381 | PE | Ongoing | Below_Median_Executed | PE_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 1385 | PA | Unfinished | Below_Median_Executed | PA_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 1402 | PE | Unfinished | AboveMedianExecuted | PE_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 1527 | PB | Ongoing | AboveMedianExecuted | PB_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 1560 | AL | Ongoing | Below_Median_Executed | AL_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 1639 | SC | Stopped | AboveMedianExecuted | SC_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 1652 | MA | Stopped | Below_Median_Executed | MA_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 1670 | MA | Unfinished | Below_Median_Executed | MA_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 1771 | AM | Ongoing | Below_Median_Executed | AM_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 1852 | MS | Ongoing | AboveMedianExecuted | MS_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 1948 | TO | Stopped | Below_Median_Executed | TO_Stopped_Below_Median_Executed | ControlCampaign | 6.666667 |
| 1952 | SP | Stopped | Below_Median_Executed | SP_Stopped_Below_Median_Executed | ControlCampaign | 6.666667 |
| 1963 | RO | Stopped | AboveMedianExecuted | RO_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 1971 | RO | Ongoing | AboveMedianExecuted | RO_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 1990 | DF | Ongoing | Below_Median_Executed | DF_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 1995 | AP | Stopped | AboveMedianExecuted | AP_Stopped_AboveMedianExecuted | Campaign | 1.176471 |
| 2022 | AC | Ongoing | AboveMedianExecuted | AC_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 2026 | AC | Ongoing | Below_Median_Executed | AC_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 2093 | CE | Stopped | Below_Median_Executed | CE_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 2108 | DF | Ongoing | Below_Median_Executed | DF_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 2206 | RO | Stopped | Below_Median_Executed | RO_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 2207 | GO | Stopped | Below_Median_Executed | GO_Stopped_Below_Median_Executed | ControlCampaign | 6.666667 |
| 2231 | ES | Ongoing | AboveMedianExecuted | ES_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 2271 | TO | Unfinished | Below_Median_Executed | TO_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 2274 | AM | Ongoing | AboveMedianExecuted | AM_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 2333 | PA | Stopped | Below_Median_Executed | PA_Stopped_Below_Median_Executed | ControlCampaign | 6.666667 |
| 2451 | RS | Unfinished | Below_Median_Executed | RS_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 2463 | DF | Ongoing | AboveMedianExecuted | DF_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 2467 | RJ | Unfinished | AboveMedianExecuted | RJ_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 2532 | CE | Ongoing | Below_Median_Executed | CE_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 2545 | TO | Ongoing | Below_Median_Executed | TO_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 2561 | RN | Ongoing | Below_Median_Executed | RN_Ongoing_Below_Median_Executed | Campaign | 1.176471 |
| 2570 | PB | Stopped | Below_Median_Executed | PB_Stopped_Below_Median_Executed | ControlCampaign | 6.666667 |
| 2571 | PB | Stopped | Below_Median_Executed | PB_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 2607 | PB | Ongoing | Below_Median_Executed | PB_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 2663 | PE | Stopped | Below_Median_Executed | PE_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 2695 | MS | Stopped | Below_Median_Executed | MS_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 2736 | PI | Stopped | Below_Median_Executed | PI_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 2791 | RO | Ongoing | Below_Median_Executed | RO_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 2847 | MT | Ongoing | Below_Median_Executed | MT_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 2862 | RJ | Ongoing | AboveMedianExecuted | RJ_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 2874 | AC | Stopped | Below_Median_Executed | AC_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 2995 | SP | Unfinished | Below_Median_Executed | SP_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 3117 | TO | Stopped | Below_Median_Executed | TO_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 3262 | SC | Unfinished | Below_Median_Executed | SC_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 3301 | MT | Unfinished | Below_Median_Executed | MT_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 3362 | PR | Unfinished | Below_Median_Executed | PR_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 3485 | RR | Ongoing | Below_Median_Executed | RR_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 3488 | RR | Stopped | AboveMedianExecuted | RR_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 3511 | ES | Stopped | Below_Median_Executed | ES_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 3533 | AP | Ongoing | AboveMedianExecuted | AP_Ongoing_AboveMedianExecuted | Campaign | 1.176471 |
| 3574 | MA | Stopped | Below_Median_Executed | MA_Stopped_Below_Median_Executed | ControlCampaign | 6.666667 |
| 3846 | SC | Stopped | Below_Median_Executed | SC_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 3866 | TO | Ongoing | AboveMedianExecuted | TO_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 3909 | ES | Unfinished | Below_Median_Executed | ES_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 3971 | AP | Ongoing | Below_Median_Executed | AP_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 4167 | MG | Stopped | Below_Median_Executed | MG_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 4527 | PE | Stopped | Below_Median_Executed | PE_Stopped_Below_Median_Executed | ControlCampaign | 6.666667 |
| 4571 | MT | Stopped | Below_Median_Executed | MT_Stopped_Below_Median_Executed | ControlCampaign | 6.666667 |
| 4656 | ES | Unfinished | Below_Median_Executed | ES_Unfinished_Below_Median_Executed | Campaign | 1.176471 |
| 4738 | PB | Stopped | AboveMedianExecuted | PB_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 4807 | CE | Stopped | Below_Median_Executed | CE_Stopped_Below_Median_Executed | ControlCampaign | 6.666667 |
| 4844 | AP | Stopped | Below_Median_Executed | AP_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 4989 | AC | Ongoing | Below_Median_Executed | AC_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 5159 | AL | Stopped | Below_Median_Executed | AL_Stopped_Below_Median_Executed | Campaign | 1.176471 |
| 5471 | AC | Stopped | AboveMedianExecuted | AC_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 6030 | RR | Unfinished | Below_Median_Executed | RR_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 6087 | RO | Unfinished | Below_Median_Executed | RO_Unfinished_Below_Median_Executed | ControlCampaign | 6.666667 |
| 6095 | SE | Ongoing | AboveMedianExecuted | SE_Ongoing_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 6118 | PI | Stopped | Below_Median_Executed | PI_Stopped_Below_Median_Executed | ControlCampaign | 6.666667 |
| 6346 | MS | Ongoing | Below_Median_Executed | MS_Ongoing_Below_Median_Executed | ControlCampaign | 6.666667 |
| 6400 | AP | Stopped | AboveMedianExecuted | AP_Stopped_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 6458 | RR | Unfinished | AboveMedianExecuted | RR_Unfinished_AboveMedianExecuted | ControlCampaign | 6.666667 |
| 6631 | AL | Stopped | Below_Median_Executed | AL_Stopped_Below_Median_Executed | ControlCampaign | 6.666667 |
| 6992 | RJ | Stopped | Below_Median_Executed | RJ_Stopped_Below_Median_Executed | ControlCampaign | 6.666667 |
| state | status | above_median_executed | var_blocking | Z_intheapp | IPW_intheapp | |
|---|---|---|---|---|---|---|
| 8 | SP | Ongoing | AboveMedianExecuted | SP_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 11 | MG | Ongoing | Below_Median_Executed | MG_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 12 | SP | Ongoing | Below_Median_Executed | SP_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 22 | RO | Ongoing | Below_Median_Executed | RO_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 51 | ES | Ongoing | Below_Median_Executed | ES_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 102 | BA | Unfinished | AboveMedianExecuted | BA_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 131 | TO | Ongoing | AboveMedianExecuted | TO_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 176 | MA | Ongoing | Below_Median_Executed | MA_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 198 | SE | Unfinished | Below_Median_Executed | SE_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 199 | PI | Unfinished | Below_Median_Executed | PI_Unfinished_Below_Median_Executed | OffApp | 6.666667 |
| 200 | PI | Unfinished | Below_Median_Executed | PI_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 201 | MG | Unfinished | AboveMedianExecuted | MG_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 233 | MG | Ongoing | AboveMedianExecuted | MG_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 270 | RN | Ongoing | AboveMedianExecuted | RN_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 272 | RN | Stopped | Below_Median_Executed | RN_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 335 | MA | Unfinished | AboveMedianExecuted | MA_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 338 | CE | Unfinished | AboveMedianExecuted | CE_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 339 | SE | Ongoing | AboveMedianExecuted | SE_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 340 | BA | Unfinished | Below_Median_Executed | BA_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 374 | MA | Ongoing | AboveMedianExecuted | MA_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 377 | MA | Ongoing | Below_Median_Executed | MA_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 397 | PA | Unfinished | AboveMedianExecuted | PA_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 401 | MS | Ongoing | AboveMedianExecuted | MS_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 419 | SP | Ongoing | Below_Median_Executed | SP_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 445 | MS | Ongoing | Below_Median_Executed | MS_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 449 | PB | Ongoing | Below_Median_Executed | PB_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 456 | CE | Stopped | AboveMedianExecuted | CE_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 459 | AL | Ongoing | AboveMedianExecuted | AL_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 460 | BA | Unfinished | AboveMedianExecuted | BA_Unfinished_AboveMedianExecuted | OffApp | 6.666667 |
| 511 | MT | Ongoing | Below_Median_Executed | MT_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 537 | AL | Unfinished | AboveMedianExecuted | AL_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 539 | MA | Unfinished | AboveMedianExecuted | MA_Unfinished_AboveMedianExecuted | OffApp | 6.666667 |
| 542 | MG | Stopped | AboveMedianExecuted | MG_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 543 | PE | Unfinished | Below_Median_Executed | PE_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 545 | TO | Stopped | AboveMedianExecuted | TO_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 558 | RS | Ongoing | AboveMedianExecuted | RS_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 559 | SC | Ongoing | Below_Median_Executed | SC_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 562 | SC | Stopped | AboveMedianExecuted | SC_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 564 | AM | Unfinished | AboveMedianExecuted | AM_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 573 | SP | Unfinished | AboveMedianExecuted | SP_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 580 | TO | Ongoing | AboveMedianExecuted | TO_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 591 | TO | Unfinished | AboveMedianExecuted | TO_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 592 | GO | Unfinished | AboveMedianExecuted | GO_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 593 | GO | Unfinished | AboveMedianExecuted | GO_Unfinished_AboveMedianExecuted | OffApp | 6.666667 |
| 597 | SC | Ongoing | AboveMedianExecuted | SC_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 605 | RN | Unfinished | AboveMedianExecuted | RN_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 624 | MS | Unfinished | AboveMedianExecuted | MS_Unfinished_AboveMedianExecuted | OffApp | 6.666667 |
| 626 | MS | Unfinished | AboveMedianExecuted | MS_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 627 | MS | Unfinished | Below_Median_Executed | MS_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 631 | MT | Ongoing | AboveMedianExecuted | MT_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 632 | MT | Ongoing | AboveMedianExecuted | MT_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 640 | SE | Unfinished | AboveMedianExecuted | SE_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 645 | ES | Ongoing | AboveMedianExecuted | ES_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 674 | SE | Ongoing | Below_Median_Executed | SE_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 676 | RS | Ongoing | Below_Median_Executed | RS_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 700 | MS | Ongoing | AboveMedianExecuted | MS_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 709 | MG | Unfinished | Below_Median_Executed | MG_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 710 | PR | Ongoing | Below_Median_Executed | PR_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 770 | RO | Stopped | AboveMedianExecuted | RO_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 776 | PE | Ongoing | Below_Median_Executed | PE_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 808 | BA | Ongoing | Below_Median_Executed | BA_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 825 | GO | Ongoing | Below_Median_Executed | GO_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 861 | AL | Ongoing | Below_Median_Executed | AL_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 868 | TO | Ongoing | Below_Median_Executed | TO_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 870 | GO | Ongoing | AboveMedianExecuted | GO_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 899 | PB | Ongoing | Below_Median_Executed | PB_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 913 | RS | Ongoing | Below_Median_Executed | RS_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 932 | RO | Ongoing | AboveMedianExecuted | RO_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 941 | ES | Ongoing | AboveMedianExecuted | ES_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 958 | CE | Ongoing | AboveMedianExecuted | CE_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 973 | PE | Ongoing | AboveMedianExecuted | PE_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 999 | GO | Ongoing | AboveMedianExecuted | GO_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 1001 | PE | Stopped | Below_Median_Executed | PE_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 1008 | AL | Stopped | AboveMedianExecuted | AL_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 1060 | MG | Ongoing | AboveMedianExecuted | MG_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 1064 | BA | Ongoing | Below_Median_Executed | BA_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 1077 | RN | Ongoing | Below_Median_Executed | RN_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 1085 | RJ | Ongoing | Below_Median_Executed | RJ_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 1091 | PB | Ongoing | AboveMedianExecuted | PB_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 1128 | RO | Ongoing | AboveMedianExecuted | RO_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 1136 | MS | Stopped | AboveMedianExecuted | MS_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 1157 | AC | Ongoing | Below_Median_Executed | AC_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 1182 | MG | Ongoing | Below_Median_Executed | MG_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 1184 | PR | Ongoing | AboveMedianExecuted | PR_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 1209 | RS | Unfinished | Below_Median_Executed | RS_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 1225 | SP | Stopped | Below_Median_Executed | SP_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 1236 | AC | Ongoing | AboveMedianExecuted | AC_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 1247 | RJ | Ongoing | Below_Median_Executed | RJ_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 1253 | RJ | Ongoing | AboveMedianExecuted | RJ_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 1284 | SE | Ongoing | Below_Median_Executed | SE_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 1296 | PI | Ongoing | AboveMedianExecuted | PI_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 1313 | PI | Ongoing | Below_Median_Executed | PI_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 1319 | PI | Ongoing | Below_Median_Executed | PI_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 1320 | PI | Ongoing | AboveMedianExecuted | PI_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 1337 | RS | Unfinished | AboveMedianExecuted | RS_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 1375 | PE | Ongoing | Below_Median_Executed | PE_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 1379 | PA | Ongoing | Below_Median_Executed | PA_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 1463 | GO | Stopped | AboveMedianExecuted | GO_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 1464 | GO | Stopped | AboveMedianExecuted | GO_Stopped_AboveMedianExecuted | OffApp | 6.666667 |
| 1508 | RO | Stopped | AboveMedianExecuted | RO_Stopped_AboveMedianExecuted | OffApp | 6.666667 |
| 1516 | RO | Ongoing | Below_Median_Executed | RO_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 1519 | ES | Ongoing | Below_Median_Executed | ES_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 1567 | CE | Ongoing | Below_Median_Executed | CE_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 1568 | CE | Ongoing | Below_Median_Executed | CE_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 1589 | BA | Ongoing | AboveMedianExecuted | BA_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 1614 | BA | Stopped | Below_Median_Executed | BA_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 1638 | SC | Ongoing | Below_Median_Executed | SC_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 1739 | PR | Ongoing | Below_Median_Executed | PR_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 1777 | AM | Ongoing | AboveMedianExecuted | AM_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 1799 | AM | Ongoing | Below_Median_Executed | AM_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 1817 | AM | Stopped | Below_Median_Executed | AM_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 1851 | MS | Ongoing | Below_Median_Executed | MS_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 1856 | TO | Ongoing | Below_Median_Executed | TO_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 1952 | SP | Stopped | Below_Median_Executed | SP_Stopped_Below_Median_Executed | OffApp | 6.666667 |
| 1953 | SP | Stopped | AboveMedianExecuted | SP_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 1992 | BA | Ongoing | AboveMedianExecuted | BA_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 1995 | AP | Stopped | AboveMedianExecuted | AP_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 1996 | AP | Stopped | AboveMedianExecuted | AP_Stopped_AboveMedianExecuted | OffApp | 6.666667 |
| 1997 | RO | Unfinished | Below_Median_Executed | RO_Unfinished_Below_Median_Executed | OffApp | 6.666667 |
| 1998 | PI | Stopped | AboveMedianExecuted | PI_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 2003 | RN | Stopped | AboveMedianExecuted | RN_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 2129 | MT | Unfinished | Below_Median_Executed | MT_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 2130 | MT | Unfinished | AboveMedianExecuted | MT_Unfinished_AboveMedianExecuted | OffApp | 6.666667 |
| 2145 | PB | Unfinished | AboveMedianExecuted | PB_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 2151 | RO | Unfinished | AboveMedianExecuted | RO_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 2172 | CE | Ongoing | AboveMedianExecuted | CE_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 2243 | MA | Ongoing | AboveMedianExecuted | MA_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 2248 | PA | Ongoing | AboveMedianExecuted | PA_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 2251 | PA | Ongoing | AboveMedianExecuted | PA_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 2255 | PE | Stopped | AboveMedianExecuted | PE_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 2256 | PB | Unfinished | AboveMedianExecuted | PB_Unfinished_AboveMedianExecuted | OffApp | 6.666667 |
| 2258 | PE | Unfinished | Below_Median_Executed | PE_Unfinished_Below_Median_Executed | OffApp | 6.666667 |
| 2262 | PI | Unfinished | AboveMedianExecuted | PI_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 2270 | TO | Unfinished | Below_Median_Executed | TO_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 2272 | AM | Unfinished | Below_Median_Executed | AM_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 2281 | MT | Stopped | AboveMedianExecuted | MT_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 2288 | MT | Stopped | Below_Median_Executed | MT_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 2294 | AM | Unfinished | Below_Median_Executed | AM_Unfinished_Below_Median_Executed | OffApp | 6.666667 |
| 2297 | AM | Ongoing | AboveMedianExecuted | AM_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 2303 | AM | Ongoing | Below_Median_Executed | AM_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 2310 | MA | Unfinished | Below_Median_Executed | MA_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 2333 | PA | Stopped | Below_Median_Executed | PA_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 2335 | PA | Stopped | AboveMedianExecuted | PA_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 2384 | SP | Ongoing | AboveMedianExecuted | SP_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 2418 | GO | Stopped | Below_Median_Executed | GO_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 2433 | SC | Ongoing | AboveMedianExecuted | SC_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 2559 | RN | Ongoing | Below_Median_Executed | RN_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 2632 | GO | Ongoing | Below_Median_Executed | GO_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 2665 | MA | Stopped | AboveMedianExecuted | MA_Stopped_AboveMedianExecuted | OffApp | 6.666667 |
| 2676 | PR | Ongoing | AboveMedianExecuted | PR_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 2695 | MS | Stopped | Below_Median_Executed | MS_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 2698 | AL | Ongoing | AboveMedianExecuted | AL_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 2736 | PI | Stopped | Below_Median_Executed | PI_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 2743 | MA | Stopped | Below_Median_Executed | MA_Stopped_Below_Median_Executed | OffApp | 6.666667 |
| 2769 | MA | Unfinished | Below_Median_Executed | MA_Unfinished_Below_Median_Executed | OffApp | 6.666667 |
| 2771 | MA | Stopped | Below_Median_Executed | MA_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 2790 | AL | Ongoing | Below_Median_Executed | AL_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 2797 | AL | Stopped | Below_Median_Executed | AL_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 2852 | PR | Stopped | AboveMedianExecuted | PR_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 2855 | PE | Ongoing | AboveMedianExecuted | PE_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 2862 | RJ | Ongoing | AboveMedianExecuted | RJ_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 2868 | AC | Stopped | AboveMedianExecuted | AC_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 2871 | AC | Ongoing | AboveMedianExecuted | AC_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 2874 | AC | Stopped | Below_Median_Executed | AC_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 2877 | BA | Stopped | AboveMedianExecuted | BA_Stopped_AboveMedianExecuted | OffApp | 6.666667 |
| 2901 | MS | Stopped | AboveMedianExecuted | MS_Stopped_AboveMedianExecuted | OffApp | 6.666667 |
| 2942 | BA | Stopped | Below_Median_Executed | BA_Stopped_Below_Median_Executed | OffApp | 6.666667 |
| 2995 | SP | Unfinished | Below_Median_Executed | SP_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 3069 | MA | Stopped | AboveMedianExecuted | MA_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 3079 | BA | Stopped | AboveMedianExecuted | BA_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 3110 | RR | Ongoing | Below_Median_Executed | RR_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 3116 | RR | Unfinished | AboveMedianExecuted | RR_Unfinished_AboveMedianExecuted | OffApp | 6.666667 |
| 3117 | TO | Stopped | Below_Median_Executed | TO_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 3128 | PA | Unfinished | AboveMedianExecuted | PA_Unfinished_AboveMedianExecuted | OffApp | 6.666667 |
| 3137 | PA | Ongoing | Below_Median_Executed | PA_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 3262 | SC | Unfinished | Below_Median_Executed | SC_Unfinished_Below_Median_Executed | OffApp | 6.666667 |
| 3278 | AC | Ongoing | Below_Median_Executed | AC_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 3293 | PR | Stopped | Below_Median_Executed | PR_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 3301 | MT | Unfinished | Below_Median_Executed | MT_Unfinished_Below_Median_Executed | OffApp | 6.666667 |
| 3360 | PR | Unfinished | Below_Median_Executed | PR_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 3390 | CE | Stopped | AboveMedianExecuted | CE_Stopped_AboveMedianExecuted | OffApp | 6.666667 |
| 3397 | RJ | Stopped | Below_Median_Executed | RJ_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 3468 | ES | Stopped | AboveMedianExecuted | ES_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 3484 | RR | Stopped | AboveMedianExecuted | RR_Stopped_AboveMedianExecuted | OffApp | 6.666667 |
| 3487 | RR | Stopped | AboveMedianExecuted | RR_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 3533 | AP | Ongoing | AboveMedianExecuted | AP_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 3534 | AP | Ongoing | Below_Median_Executed | AP_Ongoing_Below_Median_Executed | OnApp | 1.176471 |
| 3613 | CE | Unfinished | Below_Median_Executed | CE_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 3644 | RS | Ongoing | AboveMedianExecuted | RS_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 3660 | PR | Stopped | AboveMedianExecuted | PR_Stopped_AboveMedianExecuted | OffApp | 6.666667 |
| 3691 | AM | Stopped | AboveMedianExecuted | AM_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 3760 | PA | Unfinished | Below_Median_Executed | PA_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 3801 | SC | Stopped | AboveMedianExecuted | SC_Stopped_AboveMedianExecuted | OffApp | 6.666667 |
| 3842 | RR | Unfinished | Below_Median_Executed | RR_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 3846 | SC | Stopped | Below_Median_Executed | SC_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 3909 | ES | Unfinished | Below_Median_Executed | ES_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 3993 | PB | Stopped | Below_Median_Executed | PB_Stopped_Below_Median_Executed | OffApp | 6.666667 |
| 4006 | AM | Stopped | AboveMedianExecuted | AM_Stopped_AboveMedianExecuted | OffApp | 6.666667 |
| 4011 | CE | Stopped | Below_Median_Executed | CE_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 4040 | AP | Ongoing | Below_Median_Executed | AP_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 4103 | MG | Stopped | AboveMedianExecuted | MG_Stopped_AboveMedianExecuted | OffApp | 6.666667 |
| 4167 | MG | Stopped | Below_Median_Executed | MG_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 4390 | PA | Stopped | Below_Median_Executed | PA_Stopped_Below_Median_Executed | OffApp | 6.666667 |
| 4455 | RO | Stopped | Below_Median_Executed | RO_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 4494 | PA | Unfinished | Below_Median_Executed | PA_Unfinished_Below_Median_Executed | OffApp | 6.666667 |
| 4507 | RS | Stopped | Below_Median_Executed | RS_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 4511 | PB | Stopped | AboveMedianExecuted | PB_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 4540 | TO | Stopped | AboveMedianExecuted | TO_Stopped_AboveMedianExecuted | OffApp | 6.666667 |
| 4571 | MT | Stopped | Below_Median_Executed | MT_Stopped_Below_Median_Executed | OffApp | 6.666667 |
| 4580 | PA | Stopped | AboveMedianExecuted | PA_Stopped_AboveMedianExecuted | OffApp | 6.666667 |
| 4661 | AL | Unfinished | AboveMedianExecuted | AL_Unfinished_AboveMedianExecuted | OffApp | 6.666667 |
| 4672 | ES | Stopped | Below_Median_Executed | ES_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 4696 | MT | Unfinished | AboveMedianExecuted | MT_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 4718 | RS | Stopped | AboveMedianExecuted | RS_Stopped_AboveMedianExecuted | OnApp | 1.176471 |
| 4806 | AM | Unfinished | AboveMedianExecuted | AM_Unfinished_AboveMedianExecuted | OffApp | 6.666667 |
| 4844 | AP | Stopped | Below_Median_Executed | AP_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 4846 | AP | Unfinished | Below_Median_Executed | AP_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 4915 | PI | Unfinished | AboveMedianExecuted | PI_Unfinished_AboveMedianExecuted | OffApp | 6.666667 |
| 4934 | PI | Stopped | AboveMedianExecuted | PI_Stopped_AboveMedianExecuted | OffApp | 6.666667 |
| 4940 | RJ | Unfinished | Below_Median_Executed | RJ_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 4947 | GO | Unfinished | Below_Median_Executed | GO_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 4960 | PR | Unfinished | AboveMedianExecuted | PR_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 5040 | CE | Stopped | Below_Median_Executed | CE_Stopped_Below_Median_Executed | OffApp | 6.666667 |
| 5115 | TO | Unfinished | AboveMedianExecuted | TO_Unfinished_AboveMedianExecuted | OffApp | 6.666667 |
| 5329 | RR | Unfinished | AboveMedianExecuted | RR_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 5440 | RN | Ongoing | AboveMedianExecuted | RN_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 5471 | AC | Stopped | AboveMedianExecuted | AC_Stopped_AboveMedianExecuted | OffApp | 6.666667 |
| 5531 | AP | Ongoing | AboveMedianExecuted | AP_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 5565 | MT | Ongoing | Below_Median_Executed | MT_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 5571 | PB | Unfinished | Below_Median_Executed | PB_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 5573 | PB | Ongoing | AboveMedianExecuted | PB_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 5581 | PI | Stopped | Below_Median_Executed | PI_Stopped_Below_Median_Executed | OffApp | 6.666667 |
| 5600 | PR | Stopped | Below_Median_Executed | PR_Stopped_Below_Median_Executed | OffApp | 6.666667 |
| 5613 | SE | Ongoing | AboveMedianExecuted | SE_Ongoing_AboveMedianExecuted | OffApp | 6.666667 |
| 5620 | RS | Unfinished | Below_Median_Executed | RS_Unfinished_Below_Median_Executed | OffApp | 6.666667 |
| 5709 | RR | Ongoing | Below_Median_Executed | RR_Ongoing_Below_Median_Executed | OffApp | 6.666667 |
| 5791 | BA | Unfinished | Below_Median_Executed | BA_Unfinished_Below_Median_Executed | OffApp | 6.666667 |
| 5859 | PE | Unfinished | AboveMedianExecuted | PE_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 5890 | PE | Stopped | AboveMedianExecuted | PE_Stopped_AboveMedianExecuted | OffApp | 6.666667 |
| 5995 | CE | Unfinished | AboveMedianExecuted | CE_Unfinished_AboveMedianExecuted | OffApp | 6.666667 |
| 6012 | MT | Stopped | AboveMedianExecuted | MT_Stopped_AboveMedianExecuted | OffApp | 6.666667 |
| 6044 | PE | Stopped | Below_Median_Executed | PE_Stopped_Below_Median_Executed | OffApp | 6.666667 |
| 6049 | RR | Ongoing | AboveMedianExecuted | RR_Ongoing_AboveMedianExecuted | OnApp | 1.176471 |
| 6087 | RO | Unfinished | Below_Median_Executed | RO_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 6146 | PB | Stopped | Below_Median_Executed | PB_Stopped_Below_Median_Executed | OnApp | 1.176471 |
| 6401 | AP | Stopped | Below_Median_Executed | AP_Stopped_Below_Median_Executed | OffApp | 6.666667 |
| 6459 | RR | Unfinished | Below_Median_Executed | RR_Unfinished_Below_Median_Executed | OffApp | 6.666667 |
| 6514 | MG | Stopped | Below_Median_Executed | MG_Stopped_Below_Median_Executed | OffApp | 6.666667 |
| 6532 | AL | Stopped | Below_Median_Executed | AL_Stopped_Below_Median_Executed | OffApp | 6.666667 |
| 6855 | PB | Stopped | AboveMedianExecuted | PB_Stopped_AboveMedianExecuted | OffApp | 6.666667 |
| 6862 | PR | Unfinished | AboveMedianExecuted | PR_Unfinished_AboveMedianExecuted | OffApp | 6.666667 |
| 6917 | AL | Unfinished | Below_Median_Executed | AL_Unfinished_Below_Median_Executed | OnApp | 1.176471 |
| 6948 | SC | Unfinished | AboveMedianExecuted | SC_Unfinished_AboveMedianExecuted | OnApp | 1.176471 |
| 7164 | RO | Unfinished | AboveMedianExecuted | RO_Unfinished_AboveMedianExecuted | OffApp | 6.666667 |
As pre-treatment variables, we used the municipal-level characteristics below:
All the pre-treatment variables were not significant, as displayed in the main text.
Below follows the CONSORT chartflow:
Intervention 3:
Reasons for exclusion in the CONSORT:
Intervention 1: Constructions that were not in the App when the intervention started.
Intervention 2: Constructions that were not in the App when the intervention started.
Intervention 3: Constructions that e-mail contact information was not provided by the Ministry of Education website.
Statistical analysis:
For all interventions, we run the following regression model:
\[ Y_i \quad = \quad \alpha + \beta T_i + \gamma X_i + \theta F_i + \varepsilon_i \]
Where \(i\) indexes a given school observed in the intervention. \(Y_i\) represents an outcome variable. \(T_i\) represents a treatment indicator. \(\beta\) represents the estimated Average Treatment Effect. \(\gamma\) is a vector of pre-treatment coefficient effects and \(X_i\) a vector of pre-treatment covariates. \(\theta\) represents a vector of fixed effects estimands and \(F_i\) the Brazilian state level fixed effects indicator vector. \(\varepsilon_i\) is the common error term.
We run three types of analysis:
All models use municipal-level cluster robust standard errors and state-level fixed effects. For all models, we also run regressions without clustering and without fixed effects.
lfe. For the Genetic matching, we use the package MatchIt. Everything in this report is fully automated and can be reproduced using R Markdown.Further information about the workstation follows below:
sessionInfo()
## R version 3.6.0 (2019-04-26)
## Platform: x86_64-apple-darwin15.6.0 (64-bit)
## Running under: OS X El Capitan 10.11.6
##
## Matrix products: default
## BLAS: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRblas.0.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib
##
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
##
## attached base packages:
## [1] parallel stats graphics grDevices utils datasets methods
## [8] base
##
## other attached packages:
## [1] officer_0.3.5 flextable_0.5.5 rsvg_1.3
## [4] DiagrammeRsvg_0.1 DiagrammeR_1.0.1 knitr_1.23
## [7] DeclareDesign_0.18.0 fabricatr_0.8.0 ri2_0.1.2
## [10] estimatr_0.18.0 randomizr_0.18.0 MatchIt_3.0.2
## [13] stargazer_5.2.2 cobalt_3.7.0 lfe_2.8-3
## [16] Matrix_1.2-17 abjutils_0.2.3 brazilmaps_0.1.0
## [19] lubridate_1.7.4 devtools_2.1.0 usethis_1.5.1
## [22] compareGroups_4.1.0 SNPassoc_1.9-2 mvtnorm_1.0-11
## [25] survival_2.44-1.1 haplo.stats_1.7.9 haven_2.1.1
## [28] dummies_1.5.6 forcats_0.4.0 stringr_1.4.0
## [31] dplyr_0.8.3 purrr_0.3.2 readr_1.3.1
## [34] tidyr_0.8.3 tibble_2.1.3 ggplot2_3.2.0
## [37] tidyverse_1.2.1
##
## loaded via a namespace (and not attached):
## [1] uuid_0.1-2 readxl_1.3.1 backports_1.1.4
## [4] Hmisc_4.2-0 igraph_1.2.4.1 lazyeval_0.2.2
## [7] sp_1.3-1 splines_3.6.0 TH.data_1.0-10
## [10] digest_0.6.20 htmltools_0.3.6 viridis_0.5.1
## [13] magrittr_1.5 Rsolnp_1.16 checkmate_1.9.4
## [16] memoise_1.1.0 cluster_2.0.8 remotes_2.1.0
## [19] modelr_0.1.4 sandwich_2.5-1 prettyunits_1.0.2
## [22] colorspace_1.4-1 rvest_0.3.4 rgdal_1.4-4
## [25] pan_1.6 xfun_0.8 callr_3.3.0
## [28] crayon_1.3.4 jsonlite_1.6 lme4_1.1-21
## [31] brew_1.0-6 zoo_1.8-6 glue_1.3.1
## [34] kableExtra_1.1.0 gtable_0.3.0 webshot_0.5.1
## [37] MatrixModels_0.4-1 V8_2.3 pkgbuild_1.0.3
## [40] Rook_1.1-1 rms_5.1-3.1 jomo_2.6-9
## [43] SparseM_1.77 scales_1.0.0 DBI_1.0.0
## [46] Rcpp_1.0.1 viridisLite_0.3.0 xtable_1.8-4
## [49] htmlTable_1.13.1 units_0.6-3 foreign_0.8-71
## [52] Formula_1.2-3 truncnorm_1.0-8 htmlwidgets_1.3
## [55] httr_1.4.0 RColorBrewer_1.1-2 acepack_1.4.1
## [58] mice_3.6.0 pkgconfig_2.0.2 XML_3.98-1.20
## [61] nnet_7.3-12 labeling_0.3 tidyselect_0.2.5
## [64] rlang_0.4.0 visNetwork_2.0.7 munsell_0.5.0
## [67] cellranger_1.1.0 tools_3.6.0 downloader_0.4
## [70] cli_1.1.0 generics_0.0.2 broom_0.5.2
## [73] evaluate_0.14 yaml_2.2.0 processx_3.4.0
## [76] fs_1.3.1 zip_2.0.3 mitml_0.3-7
## [79] nlme_3.1-139 quantreg_5.41 xml2_1.2.0
## [82] compiler_3.6.0 rstudioapi_0.10 curl_3.3
## [85] rgexf_0.15.3 e1071_1.7-2 testthat_2.1.1
## [88] stringi_1.4.3 HardyWeinberg_1.6.3 highr_0.8
## [91] ps_1.3.0 desc_1.2.0 gdtools_0.1.9
## [94] lattice_0.20-38 classInt_0.3-3 nloptr_1.2.1
## [97] pillar_1.4.2 data.table_1.12.2 R6_2.4.0
## [100] latticeExtra_0.6-28 KernSmooth_2.23-15 gridExtra_2.3
## [103] writexl_1.1 sessioninfo_1.1.1 codetools_0.2-16
## [106] polspline_1.1.15 boot_1.3-22 MASS_7.3-51.4
## [109] assertthat_0.2.1 pkgload_1.0.2 chron_2.3-53
## [112] rprojroot_1.3-2 withr_2.1.2 multcomp_1.4-10
## [115] hms_0.4.2 influenceR_0.1.0 grid_3.6.0
## [118] rpart_4.1-15 class_7.3-15 minqa_1.2.4
## [121] rmarkdown_1.13 sf_0.7-7 base64enc_0.1-3
## [124] epitools_0.5-10
IRB: FGV exempted the primary investigator from getting IRB approval because this research mostly worked online data and was approved by the Brazilian Ministry of Education.
Pre-registration: this research has been pre-registered on the EGAP pre-registry tool: (https://egap.org/registration/4505)
Replication materials: The replication materials for this project are available here: (https://github.com/umbertomig/TDPImpactEval)
Funding: This research received funding from the Google Social Impact initiative. Google did not interfere in any aspects of the research design and analysis.
Conflict of interests: The authors of this analysis declare no conflict of interest.
##
## Percentage Concluded Before (placebo)
## ======================================================
## (1) (2) (3) (4)
## ------------------------------------------------------
## ATE -1.266 -0.536 -0.187 -0.289
## (2.353) (2.331) (2.413) (2.398)
##
## FEs No No Yes Yes
## Controls No Yes No Yes
## N 2,352 2,352 2,352 2,352
## R2 0.0002 0.014 0.029 0.040
## ======================================================
## Notes: ***Significant at the 1 percent level.
## **Significant at the 5 percent level.
## *Significant at the 10 percent level.
## Cluster-robust SEs at the Municipal Level.
The placebo test remains not significant, as expected. Note that we are using a p-value of 10%, adding extra sensitivy to the hypothesis tests carried out here.
Next, consider the percentage of construction that was carried out in this phase of the impact evaluation. The table below displays the results, presenting a null effect of the app over this indicator.
##
## Percentage Concluded
## ==================================================================
## (1) (2) (3) (4)
## ------------------------------------------------------------------
## ATE -2.721 -2.102 -0.345 -0.974
## (2.581) (2.484) (2.537) (2.482)
##
## FEs No No Yes Yes
## Controls No Yes No Yes
## N 2,352 2,352 2,352 2,352
## R2 0.001 0.023 0.044 0.053
## ==================================================================
## Notes: ***Significant at the 1 percent level.
## **Significant at the 5 percent level.
## *Significant at the 10 percent level.
## Cluster-robust Standard Errors at the Municipal Level.
We take the difference between the inicial and final percentage of the executed construction in this impact evaluation. This aims to capture changes in the construction pace, that the app could have positively impacted.
##
## Difference Percentage Begin and End
## ==================================================================
## (1) (2) (3) (4)
## ------------------------------------------------------------------
## ATE -1.455 -1.566 -0.158 -0.685
## (1.307) (1.228) (1.276) (1.261)
##
## FEs No No Yes Yes
## Controls No Yes No Yes
## N 2,352 2,352 2,352 2,352
## R2 0.001 0.046 0.062 0.084
## ==================================================================
## Notes: ***Significant at the 1 percent level.
## **Significant at the 5 percent level.
## *Significant at the 10 percent level.
## Cluster-robust Standard Errors at the Municipal Level.
A finished school in the period was labelled the schools that were still not concluded by the begining of the intervention, in August 2017, but that were finished by March 2018, when the intervention stopped. The regressions here are linear probability models, and the coefficients can be interpreted as improving the chance of XXXX.
##
## Finished Construction Indicator
## ==================================================================
## (1) (2) (3) (4)
## ------------------------------------------------------------------
## ATE 0.066*** 0.064*** 0.073*** 0.069***
## (0.008) (0.007) (0.010) (0.010)
##
## FEs No No Yes Yes
## Controls No Yes No Yes
## N 2,352 2,352 2,352 2,352
## R2 0.006 0.012 0.024 0.029
## ==================================================================
## Notes: ***Significant at the 1 percent level.
## **Significant at the 5 percent level.
## *Significant at the 10 percent level.
## Cluster-robust Standard Errors at the Municipal Level.
In this indicator, we witness a strong and consistent effect around of 5% impact. This represents that the presence of the App increased in around 5% the chance that the construction finished.
A cancelled school in the period is the schools that were still not concluded by the begining of the intervention, in August 2017, but that the construction was abandoned by March 2018, when the intervention stopped. The regressions here are linear probability models, and the coefficients can be interpreted as improving the chance of a cancellation, whenever the coefficient is positive.
##
## Cancelled Construction Indicator
## ==================================================================
## (1) (2) (3) (4)
## ------------------------------------------------------------------
## ATE 0.001 0.001 0.001 0.001
## (0.001) (0.001) (0.001) (0.001)
##
## FEs No No Yes Yes
## Controls No Yes No Yes
## N 2,352 2,352 2,352 2,352
## R2 0.0001 0.002 0.006 0.009
## ==================================================================
## Notes: ***Significant at the 1 percent level.
## **Significant at the 5 percent level.
## *Significant at the 10 percent level.
## Cluster-robust Standard Errors at the Municipal Level.
Again, the results are significant, and around 2.8%. We interpret this result as the App pressure making the policy-makers more resolute. They either work toward finish the constructions, or they report them cancelled, voiding the impact of the App.
This indicator reports whether the construction date was updated in the Federal Government databases. An update can mean a change toward a more realistic conclusion date, ameliorating the expectations of the citizens. The results below show that there were no changes in this indicator.
##
## End Date Update Indicator
## ==================================================================
## (1) (2) (3) (4)
## ------------------------------------------------------------------
## ATE 0.036 0.037 0.061 0.059
## (0.041) (0.041) (0.042) (0.041)
##
## FEs No No Yes Yes
## Controls No Yes No Yes
## N 2,352 2,352 2,352 2,352
## R2 0.001 0.016 0.061 0.076
## ==================================================================
## Notes: ***Significant at the 1 percent level.
## **Significant at the 5 percent level.
## *Significant at the 10 percent level.
## Cluster-robust Standard Errors at the Municipal Level.
##
## Percentage Concluded Before (placebo)
## ==================================================================
## (1) (2) (3) (4)
## ------------------------------------------------------------------
## ATE -0.693 -0.593 -0.625 -0.603
## (1.635) (1.611) (1.649) (1.631)
##
## FEs No No Yes Yes
## Controls No Yes No Yes
## N 2,304 2,304 2,304 2,304
## R2 0.0001 0.029 0.043 0.055
## ==================================================================
## Notes: ***Significant at the 1 percent level.
## **Significant at the 5 percent level.
## *Significant at the 10 percent level.
## Cluster-robust Standard Errors at the Municipal Level.
Placebo not significant, as expected.
##
## Percentage Concluded
## ==================================================================
## (1) (2) (3) (4)
## ------------------------------------------------------------------
## ATE -0.146 -0.025 0.029 0.046
## (1.696) (1.675) (1.712) (1.699)
##
## FEs No No Yes Yes
## Controls No Yes No Yes
## N 2,304 2,304 2,304 2,304
## R2 0.00000 0.037 0.060 0.076
## ==================================================================
## Notes: ***Significant at the 1 percent level.
## **Significant at the 5 percent level.
## *Significant at the 10 percent level.
## Cluster-robust Standard Errors at the Municipal Level.
App did not affected the percentage concluded. Moreover, the sign is flipped, suggesting that it could potentially have a negative impact on the percentage concluded.
##
## Difference Percentage Begin and End
## ==================================================================
## (1) (2) (3) (4)
## ------------------------------------------------------------------
## ATE 0.547 0.568 0.654 0.649
## (0.609) (0.612) (0.615) (0.619)
##
## FEs No No Yes Yes
## Controls No Yes No Yes
## N 2,304 2,304 2,304 2,304
## R2 0.0003 0.015 0.036 0.045
## ==================================================================
## Notes: ***Significant at the 1 percent level.
## **Significant at the 5 percent level.
## *Significant at the 10 percent level.
## Cluster-robust Standard Errors at the Municipal Level.
No effect in the difference between begin and end.
##
## Finished Construction Indicator
## ==================================================================
## (1) (2) (3) (4)
## ------------------------------------------------------------------
## ATE 0.002 0.001 0.004 0.003
## (0.015) (0.015) (0.015) (0.015)
##
## FEs No No Yes Yes
## Controls No Yes No Yes
## N 2,304 2,304 2,304 2,304
## R2 0.00001 0.011 0.024 0.031
## ==================================================================
## Notes: ***Significant at the 1 percent level.
## **Significant at the 5 percent level.
## *Significant at the 10 percent level.
## Cluster-robust Standard Errors at the Municipal Level.
No impact in the second phase of the app on the finishing of a construction.
##
## Cancelled Construction Indicator
## ==================================================================
## (1) (2) (3) (4)
## ------------------------------------------------------------------
## ATE 0.002 0.002 0.002 0.002
## (0.003) (0.003) (0.003) (0.003)
##
## FEs No No Yes Yes
## Controls No Yes No Yes
## N 2,304 2,304 2,304 2,304
## R2 0.0001 0.013 0.011 0.023
## ==================================================================
## Notes: ***Significant at the 1 percent level.
## **Significant at the 5 percent level.
## *Significant at the 10 percent level.
## Cluster-robust Standard Errors at the Municipal Level.
No effect of the App on cancelling of constructions.
##
## End Date Update Indicator
## ==================================================================
## (1) (2) (3) (4)
## ------------------------------------------------------------------
## ATE 0.000 0.000 0.000 0.000
## (0.000) (0.000) (0.000) (0.000)
##
## FEs No No Yes Yes
## Controls No Yes No Yes
## N 2,304 2,304 2,304 2,304
## ==================================================================
## Notes: ***Significant at the 1 percent level.
## **Significant at the 5 percent level.
## *Significant at the 10 percent level.
## Cluster-robust Standard Errors at the Municipal Level.
Small effect on update, in the model without Fixed effects and with controls. Very weak evidence to conclude anything concrete here.
All covariate balanced. Note that the data is not sufficient to compute the differences in the campaign for the FUNDEB transfers indicator.
##
## Percentage Concluded Before (placebo)
## ==================================================================
## (1) (2) (3) (4)
## ------------------------------------------------------------------
## ATE -1.722 -1.895 -1.842 -1.711
## (2.017) (1.997) (1.972) (1.969)
##
## FEs No No Yes Yes
## Controls No Yes No Yes
## N 1,855 1,855 1,855 1,855
## R2 0.0004 0.042 0.086 0.092
## ==================================================================
## Notes: ***Significant at the 1 percent level.
## **Significant at the 5 percent level.
## *Significant at the 10 percent level.
## Cluster-robust Standard Errors at the Municipal Level.
No effect of the campaign on the placebo.
##
## Percentage Concluded
## ==================================================================
## (1) (2) (3) (4)
## ------------------------------------------------------------------
## ATE -2.082 -2.227 -2.253 -2.075
## (2.079) (2.057) (2.018) (2.018)
##
## FEs No No Yes Yes
## Controls No Yes No Yes
## N 1,855 1,855 1,855 1,855
## R2 0.001 0.053 0.100 0.111
## ==================================================================
## Notes: ***Significant at the 1 percent level.
## **Significant at the 5 percent level.
## *Significant at the 10 percent level.
## Cluster-robust Standard Errors at the Municipal Level.
No effect of campaign on percentage concluded.
##
## Difference Percentage Begin and End
## ==================================================================
## (1) (2) (3) (4)
## ------------------------------------------------------------------
## ATE -0.360 -0.332 -0.411 -0.364
## (0.694) (0.690) (0.692) (0.691)
##
## FEs No No Yes Yes
## Controls No Yes No Yes
## N 1,855 1,855 1,855 1,855
## R2 0.0002 0.021 0.025 0.041
## ==================================================================
## Notes: ***Significant at the 1 percent level.
## **Significant at the 5 percent level.
## *Significant at the 10 percent level.
## Cluster-robust Standard Errors at the Municipal Level.
Small negative effect in the uncontrolled models. Not strong evidence enough to conclude that the campaign affected the changes in percentages reported.
##
## Finished Construction Indicator
## ==================================================================
## (1) (2) (3) (4)
## ------------------------------------------------------------------
## ATE -0.016 -0.016 -0.016 -0.015
## (0.017) (0.017) (0.017) (0.017)
##
## FEs No No Yes Yes
## Controls No Yes No Yes
## N 1,855 1,855 1,855 1,855
## R2 0.001 0.031 0.033 0.045
## ==================================================================
## Notes: ***Significant at the 1 percent level.
## **Significant at the 5 percent level.
## *Significant at the 10 percent level.
## Cluster-robust Standard Errors at the Municipal Level.
Again, very small evidence on the uncontrolled models. Suspicion: the controls have missings in particular points that make the models insignificant. Suggestion: run Amelia, or another missing data simulation device to check for the possibility of differential missing.
##
## Cancelled Construction Indicator
## ==================================================================
## (1) (2) (3) (4)
## ------------------------------------------------------------------
## ATE 0.001 0.001 0.001 0.001
## (0.001) (0.001) (0.001) (0.001)
##
## FEs No No Yes Yes
## Controls No Yes No Yes
## N 1,855 1,855 1,855 1,855
## R2 0.0002 0.002 0.003 0.005
## ==================================================================
## Notes: ***Significant at the 1 percent level.
## **Significant at the 5 percent level.
## *Significant at the 10 percent level.
## Cluster-robust Standard Errors at the Municipal Level.
Here the results are strong and robust. The campaign reduces the amount of constructions reported as cancelled by around 9%. This represents an important outcome in terms of constructions, as fewer constructions were abandoned as an effect of the campaign.
##
## End Date Update Indicator
## ==================================================================
## (1) (2) (3) (4)
## ------------------------------------------------------------------
## ATE -0.000*** -0.000*** 0.000 0.000
## (0.000) (0.000) (0.000) (0.000)
##
## FEs No No Yes Yes
## Controls No Yes No Yes
## N 1,855 1,855 1,855 1,855
## R2 -Inf.000 -Inf.000
## ==================================================================
## Notes: ***Significant at the 1 percent level.
## **Significant at the 5 percent level.
## *Significant at the 10 percent level.
## Cluster-robust Standard Errors at the Municipal Level.
Small effect, specially in the controlled and fixed effect model. However, the effect is small and we advice caution in interpreting this as an evidence that the implementer changed the dates for a better estimate.
# Visible App Begin
tab <- data.frame(table(impEvalph1$visAppIni))
names(tab) <- c('VisibleApp', 'Frequency')
tab
p<-ggplot(data=tab, aes(x=VisibleApp, y=Frequency)) +
geom_bar(stat="identity")
p
## Joining, by = "City"
## Joining, by = "City"
tab <- data.frame(table(allalert$answeredBy))
names(tab) <- c('AnsweredBy', 'Frequency')
tab
p<-ggplot(data=tab, aes(x=AnsweredBy, y=Frequency)) +
geom_bar(stat="identity")
p
# Number by construction
tab <- data.frame(table(table(allalert$project_id)))
names(tab) <- c('AlertsSameConstruction', 'Frequency')
tab
p<-ggplot(data=tab, aes(x=AlertsSameConstruction, y=Frequency)) +
geom_bar(stat="identity")
p
# Alerts Answered
tab <- data.frame(table(allalert$status))
names(tab) <- c('Status', 'Frequency')
tab
p<-ggplot(data=tab, aes(x=Status, y=Frequency)) +
geom_bar(stat="identity")
p
tab <- data.frame(table(allalert$status2))
names(tab) <- c('Status', 'Frequency')
tab
p<-ggplot(data=tab, aes(x=Status, y=Frequency)) +
geom_bar(stat="identity")
p
# Alerts by state
tab <- data.frame(sort(table(allalert$state)))
names(tab) <- c('State', 'Frequency')
tab
p<-ggplot(data=tab, aes(x=State, y=Frequency)) +
geom_bar(stat="identity") + coord_flip()
p
Barchart with frequency of alerts by each construction. This Figure shows the amount of alerts divided by the amount of schools in a given State. Increasing this number denotes that more schools are
# Alerts by state over total
tab <- data.frame(sort(table(allalert$state)))
tab2 <- data.frame(sort(table(unique(impEvalph2[,c('id_project','state')])$state)))
names(tab) <- c('State', 'Frequency')
names(tab2) <- c('State', 'Total')
tab <- left_join(tab, tab2)
## Joining, by = "State"
## Warning: Column `State` joining factors with different levels, coercing to
## character vector
tab$Proportion = 100*(tab$Frequency/tab$Total)
tab <- tab[order(tab$Proportion),]
tab$State <- factor(tab$State, levels = tab$State[order(tab$Proportion)])
tab
p<-ggplot(data=tab, aes(x=State, y=Proportion)) +
geom_bar(stat="identity") + coord_flip()
p
# Status answer by state
tab <- data.frame(table(allalert$status2, allalert$state))
names(tab) <- c('Status', 'State', 'Frequency')
tab
p<-ggplot(data=tab, aes(x=State, y=Frequency, fill=Status)) +
geom_bar(stat="identity") + coord_flip()
p
# Map not answered by state
stateCode <- pop2015[,c('ibge_code7', 'state')]
stateCode$State <- substr(stateCode$ibge_code7, 1,2)
stateCode$ibge_code7 <- NULL
stateCode <- unique(stateCode)
tab <- data.frame(table(allalert$status2, allalert$state))
tab <- tab[tab$Var1=='Not Answered',]; row.names(tab) <- NULL
names(tab) <- c('Status', 'state', 'NotAnswered')
tab <- left_join(tab, stateCode)
## Joining, by = "state"
## Warning: Column `state` joining factor and character vector, coercing into
## character vector
tab$State <- as.numeric(tab$State)
rm(stateCode)
tab$state <- NULL
tab
mapbr <- get_brmap('State')
plot_brmap(mapbr,
data_to_join = tab,
join_by = c("State" = "State"),
var = "NotAnswered") +
labs(title = 'Frequency Not Answered by State')
# Map valid answer by state
stateCode <- pop2015[,c('ibge_code7', 'state')]
stateCode$State <- substr(stateCode$ibge_code7, 1,2)
stateCode$ibge_code7 <- NULL
stateCode <- unique(stateCode)
tab <- data.frame(table(allalert$status2, allalert$state))
tab <- tab[tab$Var1=='Valid Answer',]; row.names(tab) <- NULL
names(tab) <- c('Status', 'state', 'ValidAnswer')
tab <- left_join(tab, stateCode)
## Joining, by = "state"
## Warning: Column `state` joining factor and character vector, coercing into
## character vector
tab$State <- as.numeric(tab$State)
rm(stateCode)
tab$state <- NULL
tab
mapbr <- get_brmap('State')
plot_brmap(mapbr,
data_to_join = tab,
join_by = c("State" = "State"),
var = "ValidAnswer") +
labs(title = 'Frequency Valid Answer by State')